Skip to content

fix(chat): recover saved answers and diagnose lost workers - #9120

Merged
loopx-agent merged 3 commits into
mainfrom
codex/chat-worker-recovery-diagnostics
Nov 20, 2026
Merged

loopx-agent merged 3 commits into
mainfrom
codex/chat-worker-recovery-diagnostics

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

  • Outcome basis: a saved Chat answer can be stranded if publication is interrupted. Loss of an execution worker before a full answer was also indistinguishable from a generic runtime failure in the manager's reply.
  • Before: an interrupted completion could hold the FIFO until restart or leave the original Lark receipt pending after the FIFO moved on. After: the active queue or the exact original Lark delivery can finish the saved answer once under the original execution fence. No provider rerun is needed.
  • A lost worker with no reserved answer gets a typed execution_worker_exited result. The notice describes what is known and says the request is retained and will not be replayed. Captured checkpoint I/O failures log the stage and errno without the raw exception text.
  • Intended base: loopx-project/loopx:main.

Author Declaration

  • Written by: model_agent — OpenAI Codex (GPT-6).
  • Implemented against the existing managed completion and execution-fence contracts in loopx/chat_store.py and loopx/chat_runtime.py at cc5392e9538e81487e9e551b915dabc68c085c25.
Criterion Disposition Path Validation
Preserve the saved answer and publish its transcript and terminal event once implemented ChatRuntimeController.recover_exited_turn Queue and original Lark delivery fault injection
Never recover a live worker's result or guess the effects of an incomplete request implemented Existing execution fence and terminalization Live-fence, Stop, unproven ownership, and no-provider-replay tests
Retain the same source and delivery authorization implemented LarkPrivateConversations.reconcile_request Exact original receipt readback and changed/revoked-source checks

Scope And Continuation

This repair covers publication of a reserved Chat answer and diagnosis of a lost execution worker. It doesn't reconstruct an unsaved answer or change deployment, delivery grants, or execution retry policy.

Validation

  • Tested revision: a5d05f0284d795307cf86c89b522625967847d6c.
  • Run state: finished. Input classes: synthetic and public test fixtures.
Check kind Result Public-safe evidence
regression_parity passed The new queue reproducer times out behind the original request on the unmodified baseline. It completes the original and the follow-up once each with the patch.
integration passed Injected publication and final status write failures with the real queue/store and synthetic provider fixtures. Verified one terminal reply in the original Lark receipt and one original native dispatch.
integration passed 380 focused Chat runtime, FIFO, interrupted answer, Lark delivery, and manager roundtrip regressions passed, including failure, Stop, and authorization cases.
static passed Ruff on the seven candidate paths, git diff --check, RFC pair check, and a public boundary scan on changed paths.

Provider adapters run with synthetic inputs. The live service was not restarted or used for fault injection.

Frontend / Visual Evidence

No layout or interaction change. The only visible copy change is the plain-text terminal notice for the specific lost-worker case, tested through the existing reply projection.

Type of Change

  • Bug fix
  • Test update

LoopX Area

  • Host or runtime integration

Shared-authority RFC fixture impact

N/A; existing authority and persistence contracts remain in use.

Boundary Checklist

  • Only synthetic fixtures and public-safe summaries; no private state, logs or local paths.
  • No benchmark work duplicated.
  • Scope stays within Chat answer publication and failure reporting.
  • No paired RFC changed; the exact-head RFC check passed.
  • No new UI layout or interaction.
  • DCO sign-off included.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google DeepMind · self_reported

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

[P2] 新增 execution_worker_exited 未接入控制面待处理积压(backlog)判定,导致 health().pending_count 漏计

  • 触发条件与路径:在 loopx/chat_runtime.py#L777-L778 中,ChatRuntimeController.recover_exited_turn 将同进程执行 worker 提前退出(state != LOCK_HOLDER_DEAD)的错误码从原先的 runtime_error 改为 WORKER_EXITED_CODE = "execution_worker_exited"。当该失败通知投递到 Lark 后,LarkPrivateConversations.health()(loopx/extensions/lark/private_conversations.py#L191)通过 ChatExternalConversations.quota_backlog_observer() 调用 TypeScript 控制面 planBoundConversationRequest({pending_observations})(loopx/control_plane/collaboration/conversation_binding.ts#L631-L634)判定原请求是否仍属于未获答复的待处理请求。
  • 实测反例证据:在控制面 conversation_binding.ts#L631-L634 中,允许保留为 pending: true 的失败错误码列表仅包含 runtime_error 与 server_restarted 等既有错误码(tests/test_private_chat_backlog_projection.py#L101-L107 明确验证 worker_or_preparation_failure_notice_does_not_answer_the_original_question),并未包含 execution_worker_exited。在精确提交 c68e8e20b79ded9065adafd82149b0519c605390 上通过 LarkPrivateConversations.health() 与 ChatExternalConversations.quota_backlog_pending() 构造配对反事实样本实测:runtime_error 返回 pending: True、server_restarted 返回 pending: True,而 execution_worker_exited 返回 pending: False,pending_count 由应有的 3 降为 2。
  • 实际影响:原本作为 runtime_error 保留在“待处理或回复”计数中的同进程 worker 退出失败请求,现被静默从 pending_count 中剔除,影响个人工作台与 Lark 私有连接健康度面板对未答复请求的可见性。
  • 最小修复与回归验证:在 loopx/control_plane/collaboration/conversation_binding.ts#L631-L634 的失败错误码白名单中补充 "execution_worker_exited",并在 tests/control_plane_ts/conversation_binding.test.ts 与 tests/test_private_chat_backlog_projection.py 中新增 "execution_worker_exited" 参数化回归用例,重新运行 node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts 与 pytest tests/test_private_chat_backlog_projection.py。

动机 / Motivation

受影响的调用方为使用本地 Managed Chat 队列(ChatRuntimeController)以及 Lark 私有对话投递协调路径(LarkPrivateConversations)的操作者与终端用户。

修改前,若响应已保存进入 completing 状态但在发布消息/事件或更新 completed 状态时遇到 I/O 异常,旧 Turn 会持续卡在 completing 阻塞后续 FIFO 排队,或导致原 Lark 回执长期处于未完成;同时非服务端进程死亡的 worker 退出统一退化为 runtime_error,使管家回复无法区分已保存与未保存答复的失败场景。修改后,recover_exited_turn 可在原 execution fence 下幂等完成 completing 态的剩余发布且不重跑模型,Lark 协调路径也能收尾 orphaned completing Turn,并将无保留答复的 worker 退出标记为 execution_worker_exited。

已验证队列与 Lark 原始回执在写入异常注入下仅发布一次原答复、释放后继队列且仅触发一次 turn/start;execution_worker_exited 具有专属诊断文案与脱敏 errno 日志,但该新错误码未同步接入 TypeScript 待处理积压判定导致 pending_count 统计漏计。

本 PR 不重建未保存的答复,不修改部署配置、投递授权范围或执行重试策略。

相比继续放任阻塞(无任何修改)或另建外部重跑队列,在原执行锁(execution fence)与既有 Lark 对账流内做最小幂等收尾能直接消除 FIFO 死锁风险并维持“不重跑模型”的语义。

改动思路 / Approach

本 PR 从“已保存答复的完成态发布中断”与“未保存答复的 worker 退出可观测性”两条路径切入。入口一为 ChatRuntimeController._run_started_turn(loopx/chat_runtime.py#L1127-L1193):当 update_turn(..., status="completing", response=response) 成功后设置 completion_reserved = True,若后续 update_session 或 finalize_managed_turn_completion 发生 OSError,不再误转入 _fail_turn,而是记录含 turn_id、phase=publish_saved_answer 与 exc.errno 的脱敏警告日志后返回,释放文件锁并交由队列或投递协调者收尾。入口二为 ChatRuntimeController.recover_exited_turn(loopx/chat_runtime.py#L730-L793):在获取非破坏性排他文件锁(try_exclusive_file_lock(..., preserve_holder=True))后,若确认 Turn 为托管模式且处于 completing 且无 completion_id(排除 attached host 路径),直接调用 self.store.finalize_managed_turn_completion 幂等补齐助手消息、事件与队列释放;同时 LarkPrivateConversations.reconcile_request(loopx/extensions/lark/private_conversations.py#L1922-L1925)在校验原绑定与来源后,针对 completing 状态补充调用 recover_exited_turn(session_id, turn_id),使即使在 release_active_turn 已执行但 turn.json 未能写成 completed 的窗口内,原始投递回执也能自主推进至终态并发送原答复。

恢复逻辑复用 ChatSessionStore.finalize_managed_turn_completion、execution fence(try_exclusive_file_lock)与 Lark 请求对账边界;但新增状态码 WORKER_EXITED_CODE 跨越 Python 失败写入与 TypeScript 控制面投影(conversation_binding.ts),属于需要同步跨层消费者的契约变更。

保留现有 execution fence 与单轮不自动重放原则,仅在既有 Chat 运行时与 Lark 私有对话协调路径内补全 completing 收尾与 worker 退出诊断;当前需要补齐 TypeScript pending_observations 的错误码对齐后再合并。

具体改动 / Changes

本次精确 head c68e8e20b79ded9065adafd82149b0519c605390 共修改 7 个文件(+164/-24),未涉及成对 RFC 文档变更。其中生产代码涉及 3 个文件(loopx/chat_runtime.py +29/-6、loopx/extensions/lark/manager_context.py +4/-0、loopx/extensions/lark/private_conversations.py +8/-14,含为满足模块行数上限而做的折行压缩),测试代码涉及 4 个文件(+123/-2)。没有现成独立的书面 RFC/Issue 规范,评审依据既有的 chat_store.py 与 chat_runtime.py 契约及现有控制面投影测试。

关键代码讲解

  1. ChatRuntimeController.recover_exited_turn(loopx/chat_runtime.py#L730-L793):在 try_exclusive_file_lock(path, preserve_holder=True) 确认无存活 worker 持锁并跳过 interrupting 状态后,优先判定 managed and turn and turn.get("status") == "completing" and not turn.get("completion_id") 并转交 finalize_managed_turn_completion 完成幂等发布(即使 active_turn_id 已被释放也不阻塞)。对于未进入 completing 的可恢复失败,LOCK_HOLDER_DEAD 保留 server_restarted,同进程 worker 退出则写入 WORKER_EXITED_CODE("execution_worker_exited"),并在 complete_recovered_terminalization 中保留对历史 runtime_error + WORKER_EXITED_MESSAGE 组合的兼容收尾。
  2. ChatRuntimeController._run_turn 与 _run_started_turn(loopx/chat_runtime.py#L818-L856, #L929-L1192):_run_turn 通过 phase 变量(start / execute_or_finalize)在捕获 OSError 时仅输出 turn、phase 与 exc.errno,避免泄露本地路径;_run_started_turn 在 completing 状态落盘后置 completion_reserved = True,使发布阶段遭遇 OSError 时记录 phase=publish_saved_answer 后平稳退出而非覆盖为失败。
  3. manager_failure_reply(loopx/extensions/lark/manager_context.py#L61, #L105-L107):在失败标签表与分支中新增 "execution_worker_exited",向用户明确返回“本轮执行任务在完整答复保存前退出,原请求和会话已保留。已有操作可能已生效,系统不会自动重跑这条请求”。
  4. LarkPrivateConversations.reconcile_request(loopx/extensions/lark/private_conversations.py#L1922-L1925):在 self.bindings.resolve 验证授权与来源且 turn["status"] == "completing" 时,触发 recover_exited_turn(record["session_id"], record["turn_id"]) 并重读 Turn,完成孤立 completing 态到 completed 的安全交付;其余改动为行长整理以维持文件在 2000 行上限内。

对主干的风险 / Risks

  • 积极与消极路径核验:已核验队列在发布事件遇 EIO 后自动从 completing 恢复并放行下一条排队请求、Lark 回执在 release_active_turn 后写 completed 遇 EIO 时不重跑 provider 而完成唯一投递、以及活锁持有期间调用 recover_exited_turn 不会越权发布;同时 manager_failure_reply 不会泄露原始 error 字段中的私有哨兵文本。
  • 跨层语义与积压统计回归(P2 阻塞项):虽然 steer_turn / steward_managed_mode.ts 正确地未将 execution_worker_exited 纳入自动重跑(因为操作可能已生效),且 private_web_projection.py#L300-L301 能正确复用 manager_failure_reply 显示新文案,但 loopx/control_plane/collaboration/conversation_binding.ts#L631-L634 的 pending_observations 白名单漏掉了 "execution_worker_exited",导致与改前 "runtime_error" 相比发生隐性行为漂移(pending_count 漏计)。
  • 本地验证摘要(CI 未参考 / wait_for_ci: false):
    • 聚焦回归套件(pytest 6 个相关文件共 864 项):PASS(864 passed)。
    • ruff check(7 个改动文件):PASS。
    • scripts/check_rfc_bilingual.py --base origin/main --head HEAD:PASS。
    • loopx canary premerge --from-git-diff --git-diff-base origin/main(standard 档,2 个 catalog canaries + 8 个 risk-profile smokes):PASS。
    • 控制面 TS 测试 tests/control_plane_ts/conversation_binding.test.ts(63 passed):PASS(但其枚举尚未覆盖 execution_worker_exited)。
    • 跨层反事实探测(quota_backlog_pending / LarkPrivateConversations.health()):FAIL(execution_worker_exited 判定为 pending=False,runtime_error 与 server_restarted 判定为 pending=True)。
    • mypy --follow-imports=silent(3 个生产文件):PASS(增量无新增失败)(仅存既有方法缺少返回注解的存量告警及 chat_runtime.py#L1191 裸 return)。
  • 词汇与语义对齐说明:新增本地错误码常量 WORKER_EXITED_CODE = "execution_worker_exited",目前属于 chat_runtime 与 Lark/控制面协作层的错误分类扩展,尚未同步注册到 conversation_binding.ts 的 pending_observations 集合中。

我的整体评价 / Verdict

本次提交在 completing 态幂等恢复(recover_exited_turn 与 LarkPrivateConversations.reconcile_request)、脱敏 OSError errno 日志和 execution_worker_exited 用户提示方面设计克制且本地故障注入测试充分;从长期运行与用户体验来看,本应显著降低队列卡死与误解风险,但由于未能将新错误码 execution_worker_exited 同步纳入 loopx/control_plane/collaboration/conversation_binding.ts#L631-L634 的待处理积压(backlog)判定集合,导致实际观测到 LarkPrivateConversations.health() 出现 pending_count 静默漏计回归(observable_semantics 存在未预期的行为漂移)。此外需注意 PR 描述中标注的 Tested revision 仍停留在前一提交,而实际审查的精确 head c68e8e20b79ded9065adafd82149b0519c605390 已包含后续行长整理,当前所有本地验证均已在 c68e8e20b7 上完整重跑确认。建议先补充 conversation_binding.ts 与双向回归测试后再复审。无渲染布局改动,visual_review 为不适用(仅核查 manager_context.py 纯文本回复与前端积压引用),未验证范围为无(unverified_scope: none)。

English verdict: REQUEST_CHANGES — exact head c68e8e2 breaks backlog pending-count parity because execution_worker_exited is missing from conversation_binding.ts pending_observations; local suites passed (864 pytest, premerge standard) while the backlog counterfactual failed.

@loopx-agent
loopx-agent force-pushed the codex/chat-worker-recovery-diagnostics branch from c68e8e2 to a5d05f0 Compare November 20, 2026 17:55

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google DeepMind · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用本地受管聊天队列(ChatRuntimeController)以及飞书私聊投递协调路径(LarkPrivateConversations)的操作者与个人管家用户,在对话答复已完成生成进入 completing 发布阶段遭遇 I/O 中断,或同进程执行工作线程在保存完整答复前异常退出时触发该问题。

改动前,一旦答复写入 completing 状态后在追加消息、刷新事件或写回 completed 状态时发生文件 I/O 错误,该轮次会长期滞留在 completing 阻塞整条会话 FIFO 队列,或令原始飞书投递回执永远停在未完成状态;同时同进程工作线程提前退出被笼统归为 runtime_error,无法区分已保存答复与未保存答复的退出场景。改动后,recover_exited_turn 与飞书对账入口可在原执行锁下幂等补完已保存答复的唯一发布且不重跑模型,未保存答复的工作线程退出则记为类型化错误码 execution_worker_exited 并同步纳入控制面待处理积压(backlog)统计。

注入发布阶段与终态写盘故障时,队列与飞书私聊回执均仅投递一次已保存答复并正常放行后续请求,模型仅调用一次;无保留答复的同进程退出返回明确的保守不重放提示、脱敏记录阶段与 errno,且与 runtime_error、server_restarted 一致保留在待处理计数(pending_count)中。

本 PR 不重建未保存的模型答复,不自动重放可能已产生外部副作用的未知执行,也不改变部署配置、投递授权范围或上游重试策略。

如果不做这一修复,偶发磁盘满或短暂 I/O 抖动就会把已经算完答案的对话永久卡在队列头部,后续消息全部排队超时;而直接重跑整轮又会破坏一次性副作用边界。在原执行锁与既有对账路径内补齐幂等收尾,并将新增错误码同步接入待处理积压白名单,是解决该故障的最小完整闭环。

改动思路 / Approach

整体思路是在不新增外部重跑器、不改变会话状态机主权的前提下,把“已保存答复的发布收尾”与“未保存答复的退出归因及积压投影”分层收口:

  1. 已保存答复免覆盖保护与幂等收尾:在 ChatRuntimeController._run_started_turn 中,一旦 update_turn(..., status="completing", response=response) 落盘成功即置 completion_reserved = True;此后若 update_session 或 finalize_managed_turn_completion 抛出 OSError,仅记录脱敏的 turn、phase=publish_saved_answer 与 exc.errno 并退出,绝不调用 _fail_turn 冲掉已保存答复。随后由队列推进或 LarkPrivateConversations.reconcile_request 调用 ChatRuntimeController.recover_exited_turn,在 try_exclusive_file_lock(..., preserve_holder=True) 确认原执行线程已退出且非 attached 模式(无 completion_id)后,复用 store.finalize_managed_turn_completion 完成唯一消息写入、终态事件发布与活跃轮次释放。
  2. 工作线程退出类型化归因与控制面积压对齐:对于未进入 completing 的异常退出,recover_exited_turn 区分跨进程死锁持有者(保留 server_restarted)与同进程工作线程丢失(写入 WORKER_EXITED_CODE = "execution_worker_exited"),同时在 complete_recovered_terminalization 兼容旧版 runtime_error 退出记录。本次精确提交进而在 TypeScript 控制面 planBoundConversationRequest 的 pending_observations 失败码白名单中补入 "execution_worker_exited",消除了上一提交 c68e8e20b7 中 LarkPrivateConversations.health() 漏计待处理积压的回归。
  3. 架构比例与边界判断:状态不变量留守 ChatSessionStore 与每轮文件锁,积压判定策略留守 TypeScript 控制面 conversation_binding.ts,飞书扩展仅作薄调用与文案投影,无需拆分或新增组件。当前 PR 边界严格限定于已保存答复的幂等发布收尾、同进程工作线程退出诊断与积压统计一致性;预分发阶段自动接续由独立下游切片 #9125 承接。

具体改动 / Changes

精确审查提交为 a5d05f0284d795307cf86c89b522625967847d6c(基线 origin/main 5eb6b24143bc4d421f665e95527bb332a9476313),共涉及 10 个文件(+167/-27):4 个生产文件(+42/-21)与 6 个测试文件(+125/-6),无成对双语 RFC 文件变更。仓库内无专门针对 completing 发布中断收尾与 execution_worker_exited 错误码的独立成文 RFC 或协议文档,本次依据既有会话状态机与控制面积压不变量审查。评审对照 loopx/chat_store.py、loopx/chat_runtime.py 与 loopx/control_plane/collaboration/conversation_binding.ts 的既有完成围栏、单次投递与积压投影不变量逐项核验。

关键代码讲解

  1. ChatRuntimeController.recover_exited_turn:在非破坏性排他锁下,若受管轮次处于 completing 且无 completion_id,直接委托 finalize_managed_turn_completion 幂等补完发布(即便 active_turn_id 已被释放仍可由飞书对账路径收尾);若未达 completing,按锁状态写入 server_restarted 或 execution_worker_exited,并保留 interrupting(用户 /stop)优先不被改写。
  2. ChatRuntimeController._run_turn 与 _run_started_turn:细化 phase 标记并在 completion_reserved 成立后拦截发布期 OSError,仅记录 errno 而不泄露本机路径或异常原文。
  3. LarkPrivateConversations.reconcile_request 与 manager_failure_reply:前者在通过绑定与来源核验后对孤立 completing 轮次触发 recover_exited_turn 收口(并将文件保持在 1996 行预算内);后者为 execution_worker_exited 提供明确的中文保守提示。
  4. planBoundConversationRequest:将 "execution_worker_exited" 纳入未决失败观察码列表,确保工作线程异常退出后未获实质答复的私聊请求继续计入 pending_count。

对主干的风险 / Risks

  • 正向、负向与反事实核验:
    • PASS(已保存答复幂等恢复与队列解封):在事件写出与终态更新阶段分别注入 EIO 时,原轮次与后续排队轮次均恰好完成一次,飞书回执仅收到一条终态回复,上游模型仅启动一次。
    • PASS(活锁保护与并发 /stop 边界):当工作线程仍持有活跃执行锁或轮次处于 interrupting / attached(含 completion_id)时,recover_exited_turn 严格拒绝越权收尾或覆盖取消状态。
    • PASS(上一轮 [P2] 积压漏计回归已闭环):在 a5d05f0284d795307cf86c89b522625967847d6c 上运行配对反事实探针(fixture_history(count=3) 分别注入 runtime_error、execution_worker_exited、server_restarted),LarkPrivateConversations.health() 返回 pending_count: 3, recovery_count: 0(修复前 c68e8e20b7 为 2),且底层持久化记录零副作用。
  • 本地验证矩阵(遵循 Goal 配置 wait_for_ci=false,未查询或等待远端 CI):
    • PASS:node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts(65 passed, 0 failed)与 tsc --project tsconfig.control-plane.json --noEmit(0 errors)。
    • PASS:聚焦 Python 回归套件 pytest tests/test_private_chat_backlog_projection.py tests/test_chat_queue_preparation.py tests/test_chat_runtime_activation.py tests/test_lark_private_conversations.py tests/test_steward_interrupted_answer.py(847 passed)。
    • PASS:ruff check(8 个改动 Python 文件全过)、scripts/check_rfc_bilingual.py --base origin/main --head HEAD、examples/semantic-vocabulary-drift-smoke.py、loopx check --scan-path ...(ok: true, errors: [])及 loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main(4 项直接检查 + 2 项 catalog canaries 含 maintainability ratchet 与 semantic drift 全绿,0 失败)。
  • 残余边界说明:测试在真实本地存储、文件锁与控制面进程上通过故障注入完成;未对线上运行中的守护进程直接注入故障或重启。

我的整体评价 / Verdict

精确提交 a5d05f0284d795307cf86c89b522625967847d6c 完整解决了已保存答复在发布期遇 I/O 异常卡死队列与飞书回执的缺陷,并彻底修复了上一版 c68e8e20b7 遗留的 execution_worker_exited 待处理积压漏计问题(problem_context 结论为 goal_achieved)。从结果影响看,long_horizon 为 improved(消除了无需重跑模型即可恢复的队列死锁且严守不重放未知副作用底线),user_experience 为 improved(恢复已生成答复投递、区分线程退出诊断并保持健康度积压计数准确)。视觉审查(visual_review)判定为 not_applicable:PR #9120 仅修改后端会话运行时(loopx/chat_runtime.py)、飞书私聊协调与纯文本失败提示(loopx/extensions/lark/private_conversations.py、loopx/extensions/lark/manager_context.py)以及控制面积压错误码列表(loopx/control_plane/collaboration/conversation_binding.ts),无前端页面布局或交互组件变更。 未验证范围(unverified_scope):none。代码体量克制、职责归属清晰,满足合入条件。

English verdict: APPROVE at a5d05f0. Saved-answer publication recovery, sanitized worker-exit diagnostics, and pending-backlog accounting parity are verified across 847 Python tests, 65 TypeScript tests, paired counterfactual probes, and quick premerge checks.

@loopx-agent
loopx-agent merged commit 11cc41c into main Nov 20, 2026
21 of 45 checks passed
@loopx-agent
loopx-agent deleted the codex/chat-worker-recovery-diagnostics branch November 20, 2026 18:14
loopx-agent added a commit that referenced this pull request Nov 21, 2026
* fix(cli): preserve typed runtime diagnostics in Turn failures

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): support per-endpoint model preferences and Codex-to-Antigravity failover (#9141)

Signed-off-by: LoopX Agent <[email protected]>

* Keep the prepared Chat adapter available to a retry (#9134)

A failed worker start can finish cleanup after a same-turn retry has
selected the adapter. Keep its lifetime tied to the Session and let
failed dispatch release its own worker registration. The regression
asserts one upstream turn and explicit adapter closure with the Session.

Port the fix from 4453e4bdd77486ec14b952f88ca893310b32e26c.

(cherry picked from commit 173e42b4f051b885c3a3389e1cad9c5722f7b7dd)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover zero-effect monitors and cleared-replan reports (#9143)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): admit flexible_primary_available executor transitions in conversation_scope (#9145)

Signed-off-by: LoopX Agent <[email protected]>

* Verify quota gate scope through signed lane references (#9101)

* test(quota): verify scoped gates through complete lane references

Signed-off-by: LoopX Agent <[email protected]>

* Test canonical quota gates in both public response modes

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Test C1 planning and Host input from the original quota capture (#9102)

* Use recorded quota decisions in the no-Todo settlement regression

Signed-off-by: LoopX Agent <[email protected]>

* Exercise the complete Host prompt from the captured quota source

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Prepare the CLI budget baseline with a private Git index (#9103)

Signed-off-by: LoopX Agent <[email protected]>

* Exercise C1 Goal delivery and bound replan from the saved source (#9104)

Signed-off-by: LoopX Agent <[email protected]>

* Keep long claim overlap scopes on demand (#9086)

* Link long thin-claim overlap lists to their receipts

Signed-off-by: LoopX Agent <[email protected]>

* Read claim overlap details without copying the lease

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): preserve explicit Monitor Todo selection (#9099)

Carry Todo selection provenance into quota admission. Preserve explicit secondary Monitor polling and recovery from previously admitted receipts.

Signed-off-by: LoopX Agent <[email protected]>

* Keep ready Todo scope details on the required context read (#9106)

Signed-off-by: LoopX Agent <[email protected]>

* Fix archive rollback with custom install directories (#9066)

* refactor(core): defer benchmark implementations during capability discovery

Preserve the benchmark toolkit public exports and static types while keeping execution and experiment implementations out of metadata discovery. Qualify source and installed Core paths, and reconcile the paired minimal/Standard RFC boundary without changing activation or authority defaults.

Signed-off-by: cicolord <[email protected]>

* Defer Yardstick evaluation until an enabled Explore read

Signed-off-by: cicolord <[email protected]>

* Omit empty Todo summary lanes from ordinary Worker context

Signed-off-by: cicolord <[email protected]>

* fix(update): honor configured archive directories during rollback

Signed-off-by: LoopX Agent <[email protected]>

* fix(lark): replay recall guard on current main

Signed-off-by: huangruiteng <[email protected]>

* docs: record the verified LoopX 2.3.0 release baseline

Signed-off-by: LoopX Agent <[email protected]>

* Preserve quota spend source when prepared work is admitted

Signed-off-by: LoopX Agent <[email protected]>

* test(quota): qualify native saved complete-Turn checkpoints

Signed-off-by: LoopX Agent <[email protected]>

* Retry cleanup of an owned file mutation marker

Signed-off-by: LoopX Agent <[email protected]>

* Keep synthetic Git-path test data outside the public source scan

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): share native preparation wait with readiness

Signed-off-by: LoopX Agent <[email protected]>

* fix(doctor): bind package update readback to its Python environment

Signed-off-by: LoopX Agent <[email protected]>

* Keep Doctor discovery separate from service activation

Signed-off-by: LoopX Agent <[email protected]>

* Verify the recorded Python launcher before Doctor probes it

Signed-off-by: LoopX Agent <[email protected]>

* fix(desktop): require runtime qualification for macOS preview updates

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): reuse replan projection owners during recovery

Signed-off-by: LoopX Agent <[email protected]>

* Qualify Codex 0.169 native context replay on main

Signed-off-by: LoopX Agent <[email protected]>

* Stop regenerating retired quota view aliases

Signed-off-by: LoopX Agent <[email protected]>

* fix: pin deferred native context reads to the original capture

Signed-off-by: LoopX Agent <[email protected]>

* Keep native capture navigation within small output budgets

Signed-off-by: LoopX Agent <[email protected]>

* Retire legacy heartbeat work prompt generation

Signed-off-by: LoopX Agent <[email protected]>

* Capture C1 composition failures as receipt diagnostics

Signed-off-by: LoopX Agent <[email protected]>

* Recover interrupted native Todo readiness from the original Turn

Signed-off-by: LoopX Agent <[email protected]>

* Revalidate the original lease before publishing resumed native context

Signed-off-by: LoopX Agent <[email protected]>

* Fence native readiness publication with the canonical lease writer

Signed-off-by: LoopX Agent <[email protected]>

* fix(scheduler): keep Codex fallback on the original C1 runtime

Signed-off-by: LoopX Agent <[email protected]>

* fix(scheduler): pin the originating fallback launchers

Signed-off-by: LoopX Agent <[email protected]>

* Retire the duplicate Codex App scheduler projection

Signed-off-by: LoopX Agent <[email protected]>

* Seed the recorded scheduler profile in the elapsed-backoff fixture

Signed-off-by: LoopX Agent <[email protected]>

* Verify canonical cadence on refused and settled Turns

Signed-off-by: LoopX Agent <[email protected]>

* Pin named Core commands in saved scheduler fallbacks

Signed-off-by: LoopX Agent <[email protected]>

* fix(research): preserve saved case addresses when a version cannot open

Signed-off-by: LoopX Agent <[email protected]>

* Preserve citation focus after a saved study refresh

Signed-off-by: LoopX Agent <[email protected]>

* Capture authenticated LoopX receipts for Research attention (#8790)

* feat(research): capture authenticated LoopX attention receipts

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* fix(research): tighten attention receipt provenance

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* fix(research): publish attention captures after source validation

Signed-off-by: huangruiteng <[email protected]>

---------

Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>
Co-authored-by: huangruiteng <[email protected]>

* Recover the original Session and Turn from a confirmed Chat task

Signed-off-by: huangruiteng <[email protected]>

* perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops (#9041)

* perf(lark,native_chat): avoid full directory reparsing in steward chat loops

- Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture.
- Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors.
- Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated.
- Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py.

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): project small recovery metadata fields to stop data_url memory growth

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* Retire transitional smoke checks and duplicate execution (#9073)

Signed-off-by: lusendong.6789 <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* feat(steward): support probe-gated generic continuous_monitor todos with handoff and user actions (#9074)

Signed-off-by: LoopX Agent <[email protected]>

* Remove redundant defensive prose from prompts and comments (#9072)

Remove redundant defensive prose while preserving published heartbeat wrappers and externally managed skill readiness.

Validated with 197 related tests, independent historical wrapper migrations and negative cases, installed-skill readback, and current-main quick canary. Existing CI qualification and infrastructure limitations remain recorded in PR #9072.

Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>

* Keep original Chat retry errors visible during receipt refresh

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): tolerate report validator python drift and allow cancel of mid-turn failed reports (#9077)

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): add exact-head PR review signals and protected merge confirmation (#9081)

Signed-off-by: LoopX Agent <[email protected]>

* feat(steward): add isolated OpenViking long-term memory with per-speaker boundaries (#9083)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(content-ops): slim surface module and eliminate Any (#8961)

* refactor(content-ops): slim surface module and eliminate Any

Signed-off-by: LoopX Agent <[email protected]>

* fix(content-ops): reject malformed angle source_item_ids containers

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(collaboration): batch handoff query Goal lifetime guard lock per Goal (#9085)

Signed-off-by: LoopX Agent <[email protected]>

* fix(hosts): raise managed service startup readiness timeout to 45s (#9087)

Signed-off-by: LoopX Agent <[email protected]>

* perf(manager-context): memoize history_inspect lifecycle checks and scope runtime revision in handoffs query (#9088)

Eliminate the remaining O(N) Effect runtime RPCs and uncached runtime fingerprinting inside tracking.query(): enter effect_runtime_request_scope() for the query, memoize pure history_inspect lifecycle decisions across entries sharing identical Goal refs while holding the Goal lifetime guard, lazily discover legacy ingress files only when external audience recovery needs them, and preserve entry scan order across Goal batches.

Signed-off-by: LoopX Agent <[email protected]>

* refactor(lark): slim private_conversations below 2000 lines with net LOC reduction (#8929)

Signed-off-by: LoopX Agent <[email protected]>

* perf(chat,hosts): skip frontend source re-hashing on immutable installed releases and extend readiness timeout to 90s (#9089)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(control-plane): split and slim quota monitor_poll_commit (#8885)

* refactor(control-plane): split and slim quota monitor_poll_commit

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): preserve legacy lane budget and observation diagnostic order

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* refactor(quota): slim loopx/quota.py and tighten Any count to zero (#8939)

Signed-off-by: LoopX Agent <[email protected]>

* refactor(native-chat): slim external_conversations below 2k lines and retire ceiling (#8911)

* refactor(native-chat): move steward commission mixin to steward_commands and retire external_conversations ceiling

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): respect include_raw_response in retained result recovery

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* refactor(chat): slim chat_runtime below 1,500 lines, fix entry-point project_contexts guard, and retire ceiling (#8920)

* refactor(chat): move session lifecycle mixin to session_open, deduplicate runtime helpers, and retire ceiling

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): drop turn_done_events slot before waking waiters on Thread.start failure

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans (#9093)

Signed-off-by: LoopX Agent <[email protected]>

* perf(lark,native_chat): back off noop recovery retries and cache recovery lookups (#9096)

Signed-off-by: LoopX Agent <[email protected]>

* perf(native_chat,lark): cache unchanged quota backlog health and interrupt observations (#9098)

Signed-off-by: LoopX Agent <[email protected]>

* perf(lark,native_chat): coalesce listener snapshots and cache recovery event summaries (#9100)

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: cicolord <[email protected]>
Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>
Co-authored-by: cicolord <[email protected]>
Co-authored-by: huangruiteng <[email protected]>
Co-authored-by: Lusen Dong <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>

* refactor(issue-fix): consolidate maintainer-correction admission and slim pr_lifecycle (#8897)

Signed-off-by: LoopX Agent <[email protected]>

* Fix truncated requests and responses in the Codex API relay (#8713)

* fix(codex relay): fence incomplete model requests before drain

Signed-off-by: LoopX Agent <[email protected]>

* fix(codex relay): surface incomplete upstream responses

Port the verified response framing fix to the request-admission branch. Original candidate 27bc3aee311cdfc6a00466ff9d3b58c7db36e01b.

Signed-off-by: LoopX Agent <[email protected]>

* Bound Codex relay writes to clients that stop reading

Signed-off-by: LoopX Agent <[email protected]>

* Reject upstream chunked responses missing the trailer terminator

Signed-off-by: LoopX Agent <[email protected]>

* Validate upstream chunk data delimiters before relay completion

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): repair dsh closeout test, antigravity failover alias, and release download retry (#9147)

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): update v2.4.0 release notes and dsh-loopx-plugin 0.1.1-beta.7 README (#9149)

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): recover saved answers and diagnose lost workers (#9120)

* fix(chat): recover saved answers and diagnose lost workers

Signed-off-by: LoopX Agent <[email protected]>

* style(lark): keep private_conversations within module line budget

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): retain execution_worker_exited in pending backlog count

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(chat): resume steward requests lost before dispatch (#9125)

Signed-off-by: LoopX Agent <[email protected]>

* fix(benchmark): restore frozen v1 archives and align checkpoint/runtime tests with C1 defaults (#9151)

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): stabilize CLI help, architecture guards, onboarding gate, and public smokes (#9152)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover report selection across python upgrades and prevent queue kill on observation monitors (#9155)

Signed-off-by: LoopX Agent <[email protected]>

* fix(release): align extensions, chat, presentation, and transport tests for v2.4.0 (#9156)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice (Slice B1) (#9154)

* fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice

Signed-off-by: LoopX Agent <[email protected]>

* fix(quota): preserve tokenless identity and exact numeric types in attach_complete_turn_context

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover frozen report contract failures and rebind resumed task leases (#9157)

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): recover pre-dispatch runtime failures across endpoints and stop receipt-unknown disk churn (#9159)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control_plane): align lease recovery, settlement readback, and runtime fixtures (#9158)

* fix(control_plane): align lease recovery, settlement readback, and runtime fixtures

Signed-off-by: LoopX Agent <[email protected]>

* fix(control_plane): forward record_capture in worker outage test and harden version type guard

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 explore, context, and checkpoint tail slice (Slice B4) (#9161)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): stabilize v2.4.0 replan, monitor recovery, and staged-read slice (Slice B5) (#9160)

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording (#9162)

* docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording

Signed-off-by: LoopX Agent <[email protected]>

* docs(release): align #9158 and #9161 descriptions with verified commits

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>

* fix(steward): tolerate minor JSON delimiter glitches and recover saved report candidates (#9164)

Signed-off-by: LoopX Agent <[email protected]>

* fix(control-plane): preserve bound yardstick reversals in native replan openings (#9165)

Signed-off-by: LoopX Agent <[email protected]>

* fix(native-chat): recover loopback relay 401 failures and isolate test auth (#9167)

Signed-off-by: LoopX Agent <[email protected]>

---------

Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: cicolord <[email protected]>
Signed-off-by: huangruiteng <[email protected]>
Signed-off-by: lusendong.6789 <[email protected]>
Signed-off-by: Lusen Dong <[email protected]>
Co-authored-by: cicolord <[email protected]>
Co-authored-by: huangruiteng <[email protected]>
Co-authored-by: Lusen Dong <[email protected]>
Co-authored-by: TRAE CLI <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant